Compare Machine Learning Models Research Articles

MLCompare: Platform to Compare Machine Learning Models

Abstract: The field of machine learning (ML) continues to advance, the demand for efficient and effective methods for model development and selection becomes increasingly pressing. Automated Machine Learning (AutoML) platforms have emerged as powerful tools to address this need by automating the process of model selection, hyperparameter tuning, and feature engineering. By integrating a wide array of ML techniques and leveraging cloud computing resources, the platform enables researchers and practitioners to efficiently explore and compare the performance of various algorithms across diverse datasets. Key features include a user-friendly interface, a robust evaluation framework encompassing traditional metrics and advanced techniques, and parallel processing capabilities. Through experiments on benchmark datasets from multiple domains, the platform demonstrates its ability to streamline algorithm selection and provide valuable insights into algorithm performance. This contribution advances the field of AutoML by facilitating informed decision-making and accelerating innovation in ML applications.

Optimized Approach for Heart Disease Prediction by using Enhancing Diagnostic Method through Meta-Features

In today's world, pollution is worsening, and bad habits like not eating regularly, eating lots of junk food, and not exercising enough are becoming more common. This can cause health problems, and the global need for the detection of heart diseases is increasing. The American Heart Association, in 2023, has given an update that heart disease is the leading cause of death. Implementing machine learning models has given significant results, but due to the limitations of requirement balanced data, repeat training model complexity has led to unreliable results in some cases. Hence, the proposed model overcomes the limitations by comparing machine learning models for cardiovascular diseases. Performance of proposed work is evaluated with the base models KNN and DT with and without smooth. Thus, the comparison included increased accuracy in DT, but the proposed model GBM as a meta learner has led the performance metric with 92% accuracy, with recall of 0.89% and F1 score of 0.86%. Thus, the proposed approach has achieved the highest accuracy in cardiovascular diseases using meta-subjects. Future developments for the research will focus on applying the model to a larger dataset and analysing cases according to complex machine learning models.

Quantitative analysis of microplastics in water environments based on Raman spectroscopy and convolutional neural network

With the increasing interest in microplastics (MPs) pollutants, quantitative analysis of MPs in water environment is an important issue. Vibrational spectroscopy, represented by Raman spectroscopy, is widely used in MP detection because they can provide unique fingerprint characteristics of chemical components of MPs, but it is difficult to provide quantitative information. In this paper, an ingenious method for quantitative analysis of MPs in water environment by combining Raman spectroscopy and convolutional neural network (CNN) is proposed. It is innovatively proposed to collect the average mapping spectra (AMS) of the samples to improve the uniformity of Raman spectroscopy detection, and to increase the effective detection range of concentration by filtering different volumes of the same MP solutions. In order to verify the universality and effectiveness of the proposed method, 6 different sizes of Polyethylene (PE) MPs were used as detection objects and mixed into 5 different actual water environments. The R2 and RMSE of CNN for identifying the concentration of PE solutions could reach 0.9972 and 0.033, respectively. Meanwhile, by comparing machine learning models such as Random Forest (RF) and Support Vector Machine (SVM) were compared, and CNN combined with Raman spectroscopy has significant advantages in identifying the concentration of MPs.

The features associated with mammography-occult MRI-detected newly diagnosed breast cancer analysed by comparing machine learning models with a logistic regression model.

To compare machine learning (ML) models with logistic regression model in order to identify the optimal factors associated with mammography-occult (i.e. false-negative mammographic findings) magnetic resonance imaging (MRI)-detected newly diagnosed breast cancer (BC). The present single-centre retrospective study included consecutive women with BC who underwent mammography and MRI (no more than 45days apart) for breast cancer between January 2018 and May 2023. Various ML algorithms and binary logistic regression analysis were utilized to extract features linked to mammography-occult BC. These features were subsequently employed to create different models. The predictive value of these models was assessed using receiver operating characteristic curve analysis. This study included 1957 malignant lesions from 1914 patients, with an average age of 51.64 ± 9.92years and a range of 20-86years. Among these lesions, there were 485 mammography-occult BCs. The optimal features of mammography-occult BC included calcification status, tumour size, mammographic density, age, lesion enhancement type on MRI, and histological type. Among the different ML models (ANN, L1-LR, RF, and SVM) and the LR-based combined model, the ANN model with RF features was found to be the optimal model. It demonstrated the best discriminative performance in predicting mammography false- negative findings, with an AUC of 0.912, an accuracy of 86.90%, a sensitivity of 85.85%, and a specificity of 84.18%. Mammography-occult MRI-detected breast cancers have features that should be considered when performing breast MRI to improve the detection rate for breast cancer and aid in clinician management.

Biosignal comparison for autism assessment using machine learning models and virtual reality

Clinical assessment procedures encounter challenges in terms of objectivity because they rely on subjective data. Computational psychiatry proposes overcoming this limitation by introducing biosignal-based assessments able to detect clinical biomarkers, while virtual reality (VR) can offer ecological settings for measurement. Autism spectrum disorder (ASD) is a neurodevelopmental disorder where many biosignals have been tested to improve assessment procedures. However, in ASD research there is a lack of studies systematically comparing biosignals for the automatic classification of ASD when recorded simultaneously in ecological settings, and comparisons among previous studies are challenging due to methodological inconsistencies. In this study, we examined a VR screening tool consisting of four virtual scenes, and we compared machine learning models based on implicit (motor skills and eye movements) and explicit (behavioral responses) biosignals. Machine learning models were developed for each biosignal within the virtual scenes and then combined into a final model per biosignal. A linear support vector classifier with recursive feature elimination was used and tested using nested cross-validation. The final model based on motor skills exhibited the highest robustness in identifying ASD, achieving an AUC of 0.89 (SD = 0.08). The best behavioral model showed an AUC of 0.80, while further research is needed for the eye-movement models due to limitations with the eye-tracking glasses. These findings highlight the potential of motor skills in enhancing objectivity and reliability in the early assessment of ASD compared to other biosignals.

The role of hyperparameters in machine learning models and how to tune them

Abstract Hyperparameters critically influence how well machine learning models perform on unseen, out-of-sample data. Systematically comparing the performance of different hyperparameter settings will often go a long way in building confidence about a model's performance. However, analyzing 64 machine learning related manuscripts published in three leading political science journals (APSR, PA, and PSRM) between 2016 and 2021, we find that only 13 publications (20.31 percent) report the hyperparameters and also how they tuned them in either the paper or the appendix. We illustrate the dangers of cursory attention to model and tuning transparency in comparing machine learning models’ capability to predict electoral violence from tweets. The tuning of hyperparameters and their documentation should become a standard component of robustness checks for machine learning models.

Machine learning to identify chronic cough from administrative claims data

Accurate identification of patient populations is an essential component of clinical research, especially for medical conditions such as chronic cough that are inconsistently defined and diagnosed. We aimed to develop and compare machine learning models to identify chronic cough from medical and pharmacy claims data. In this retrospective observational study, we compared 3 machine learning algorithms based on XG Boost, logistic regression, and neural network approaches using a large claims and electronic health record database. Of the 327,423 patients who met the study criteria, 4,818 had chronic cough based on linked claims–electronic health record data. The XG Boost model showed the best performance, achieving a Receiver-Operator Characteristic Area Under the Curve (ROC-AUC) of 0.916. We selected a cutoff that favors a high positive predictive value (PPV) to minimize false positives, resulting in a sensitivity, specificity, PPV, and negative predictive value of 18.0%, 99.6%, 38.7%, and 98.8%, respectively on the held-out testing set (n = 82,262). Logistic regression and neural network models achieved slightly lower ROC-AUCs of 0.907 and 0.838, respectively. The XG Boost and logistic regression models maintained their robust performance in subgroups of individuals with higher rates of chronic cough. Machine learning algorithms are one way of identifying conditions that are not coded in medical records, and can help identify individuals with chronic cough from claims data with a high degree of classification value.

Multimodal imaging measures in the prediction of clinical response to deep brain stimulation for refractory depression: A machine learning approach

Objectives This study compared machine learning models using unimodal imaging measures and combined multi-modal imaging measures for deep brain stimulation (DBS) outcome prediction in treatment resistant depression (TRD). Methods Regional brain glucose metabolism (CMRGlu), cerebral blood flow (CBF), and grey matter volume (GMV) were measured at baseline using 18F-fluorodeoxy glucose (18F-FDG) positron emission tomography (PET), arterial spin labelling (ASL) magnetic resonance imaging (MRI), and T1-weighted MRI, respectively, in 19 patients with TRD receiving subcallosal cingulate (SCC)-DBS. Responders (n = 9) were defined by a 50% reduction in HAMD-17 at 6 months from the baseline. Using an atlas-based approach, values of each measure were determined for pre-selected brain regions. OneR feature selection algorithm and the naïve Bayes model was used for classification. Leave-out-one cross validation was used for classifier evaluation. Results The performance accuracy of the CMRGlu classification model (84%) was greater than CBF (74%) or GMV (74%) models. The classification model using the three image modalities together led to a similar accuracy (84%0 compared to the CMRGlu classification model. Conclusions CMRGlu imaging measures may be useful for the development of multivariate prediction models for SCC-DBS studies for TRD. The future of multivariate methods for multimodal imaging may rest on the selection of complementing features and the developing better models.Clinical Trial Registration: ClinicalTrials.gov (#NCT 01983904)

Improving urban water demand forecast using conformal prediction-based hybrid machine learning models

This paper presents a probabilistic forecasting method that predicts the future water consumption patterns, taking into account the inherent uncertainty in the system. The proposed method leverages statistical techniques to estimate the probability distribution of future water demand scenarios. The findings of this study will provide decision-makers with a range of possible alternatives that facilitate better planning and management of water resources. We developed a conformal prediction-based hybrid demand forecast model, which combines convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) while comparing it with other machine learning models for probabilistic hourly water demand forecasting. Additionally, we address crucial considerations when implementing a probabilistic forecasting system, including selecting appropriate data and choosing model parameters. The performance of the proposed model is validated for probabilistic water demand forecasting in real-world settings. Results indicate noteworthy improvement in deterministic and probabilistic predictions by 10 % and 26.7 %, respectively. The findings underscore the potential benefits of this approach for improved decision-making and resource management. The results also demonstrate that the proposed hybrid model outperforms the compared machine learning models with better prediction performance and increased robustness for handling prediction uncertainties.

Comparing Machine Learning Models to Determine the Effect of Speech Duration on Speaker Identification within Kazakh Speech Corpus

This paper conducts comparative analysis of speaker identification techniques, namely Gaussian Mixture Models (GMM), Bidirectional Long Short-Term Memory (BiLSTM) neural networks, and Recurrent Neural Networks (RNN), applied to the Kazakh Speech Corpus. The study's central objective is to explore the influence of speech duration on the efficacy of speaker identification. Our results indicate that all models exhibit enhanced accuracy with increased speech duration. Furthermore, the incorporation of delta and fundamental frequency (F0) as supplementary features bolsters the performance of these models. This research contributes to the understanding of how speech duration and additional features can optimize speaker identification techniques, providing valuable insights for future developments in the field.

A prognostic index to predict symptom and functional outcomes of a coached, web-based intervention for trauma-exposed veterans.

Researchers at the Department of Veterans Affairs (VA) have studied interventions for posttraumatic stress disorder and co-occurring conditions in both traditional and digital formats. One such empirically supported intervention is web skills training in affective and interpersonal regulation (webSTAIR), a coached, 10-module web program based on STAIR. To understand which patient characteristics were predictive of webSTAIR outcomes in a sample of trauma-exposed veterans (N = 189), we used machine learning (ML) to develop a prognostic index from among 18 baseline characteristics (i.e., demographic, military, trauma history, and clinical) to predict posttreatment posttraumatic stress disorder severity, depression severity, and psychosocial functioning impairment. We compared the ML models to a benchmark of linear regression models in which the only predictor was the baseline severity score of the outcome measure. The ML and "severity-only" models performed similarly, explaining 39%-45% of the variance in outcomes. This suggests that baseline symptom severity and functioning are strong indicators for webSTAIR outcomes in veterans, with higher severity indicating worse prognosis, and that the other variables examined did not contribute significant added predictive signal. Findings also highlight the importance of comparing ML models to an appropriate benchmark. Future research with larger samples could potentially detect smaller patient-level effects as well as effects driven by other types of variables (e.g., therapeutic process variables). As a transdiagnostic, digital intervention, webSTAIR can potentially serve a diverse veteran population with varying trauma histories and may be best conceptualized as a beneficial first step of a stepped care model for those with heightened symptoms or impairment. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Improving preoperative prediction of surgery duration

BackgroundOperating rooms (ORs) are one of the costliest units in a hospital, therefore the cumulative consequences of any kind of inefficiency in OR management lead to a significant loss of revenue for the hospital, staff dissatisfaction, and patient care disruption. One of the possible solutions to improving OR efficiency is knowing a reliable estimate of the duration of operations. The literature suggests that the current methods used in hospitals, e.g., a surgeon’s estimate for the given surgery or taking the average of only five previous records of the same procedure, have room for improvement.MethodsWe used over 4 years of elective surgery records (n = 52,171) from one of the major metropolitan hospitals in Australia. We developed robust Machine Learning (ML) approaches to provide a more accurate prediction of operation duration, especially in the absence of surgeon’s estimation. Individual patient characteristics and historic surgery information attributed to medical records were used to train predictive models. A wide range of algorithms such as Extreme Gradient Boosting (XGBoost) and Random Forest (RF) were tested for predicting operation duration.ResultsThe results show that the XGBoost model provided statistically significantly less error than other compared ML models. The XGBoost model also reduced the total absolute error by 6854 min (i.e., about 114 h) compared to the current hospital methods.ConclusionThe results indicate the potential of using ML methods for reaching a more accurate estimation of operation duration compared to current methods used in the hospital. In addition, using a set of realistic features in the ML models that are available at the point of OR scheduling enabled the potential deployment of the proposed approach.

An Exploration of Flow Control Using Machine Learning and Computational Fluid Dynamics

Although numerous studies relating to computational fluid dynamics and machine learning have been conducted in relation to automotive development, the majority focus on either early development using completed 3D models, or the final testing stages of development, or machine learning accelerated computational fluid dynamic simulations. While this approach is helpful in software development and simulation, it is not easily adaptable to automotive design where the final model is constantly changing and being modified. Consequently, the aim of this study is to propose a method for conducting computational fluid dynamics and machine learning concurrently to accelerate the development process. The proposed method is used to design and improve the aerodynamic efficiency of an object. The approach focuses on developing, implementing, and comparing machine learning models capable of generating optimised three-dimensional objects with the required geometry to direct airflow paths required in applications such as pressure generation, as needed for both active and passive flow control. The study concludes that both decision tree regression and long short-term memory (LSTM) autoencoder models could be used to optimise the aerodynamic efficiency of solid bodies, but that the LSTM autoencoder performs better overall. An undesirable effect of the shape optimisation is an overall reduction in shape size as optimization increases.

Comparative Analysis of Machine Learning Models for Bankruptcy Prediction in the Context of Pakistani Companies

This article presents a comparative analysis of machine learning models for business failure prediction. Bankruptcy prediction is crucial in assessing financial risks and making informed decisions for investors and regulatory bodies. Since machine learning techniques have advanced, there has been much interest in predicting bankruptcy due to their capacity to handle complex data patterns and boost prediction accuracy. In this study, we evaluated the performance of various machine learning algorithms. We collect comprehensive data comprising financial indicators and company-specific attributes relevant to the Pakistani business landscape from 2016 through 2021. The analysis includes AdaBoost, decision trees, gradient boosting, logistic regressions, naive Bayes, random forests, and support vector machines. This comparative analysis provides insights into the most suitable model for accurate bankruptcy prediction in Pakistani companies. The results contribute to the financial literature by comparing machine learning models tailored to anticipate Pakistani stock market insolvency. These findings can assist financial institutions, regulatory bodies, and investors in making more informed decisions and effectively mitigating financial risks.

Detecting future financial statement fraud using a machine learning model in Indonesia: a comparative study

PurposeThis study aims to compare machine learning models, datasets and splitting training-testing using data mining methods to detect financial statement fraud.Design/methodology/approachThis study uses a quantitative approach from secondary data on the financial reports of companies listed on the Indonesia Stock Exchange in the last ten years, from 2010 to 2019. Research variables use financial and non-financial variables. Indicators of financial statement fraud are determined based on notes or sanctions from regulators and financial statement restatements with special supervision.FindingsThe findings show that the Extremely Randomized Trees (ERT) model performs better than other machine learning models. The best original-sampling dataset compared to other dataset treatments. Training testing splitting 80:10 is the best compared to other training-testing splitting treatments. So the ERT model with an original-sampling dataset and 80:10 training-testing splitting are the most appropriate for detecting future financial statement fraud.Practical implicationsThis study can be used by regulators, investors, stakeholders and financial crime experts to add insight into better methods of detecting financial statement fraud.Originality/valueThis study proposes a machine learning model that has not been discussed in previous studies and performs comparisons to obtain the best financial statement fraud detection results. Practitioners and academics can use findings for further research development.

Computationally constrained audio-based violence detection through transfer learning and data augmentation techniques

Audio-based violence detection is a critical research area for enhancing public safety and security. This paper delves into comparing machine learning models, specifically Convolutional Neural Networks and Shallow Networks in the context of audio violence detection. We evaluate these models under varying training set configurations and data augmentation techniques, analyzing their impact on model performance and robustness under varying real-world conditions. Specifically, we address the issue of domain shifts, exploring how models perform under different types of noise and reverberation. Our results highlight scenarios where Shallow Networks, despite their lower computational costs, exhibit performance nearly on par with that of high-cost CNNs. Introducing tailored data augmentation techniques significantly enhances the models' performance and stability against domain shifts, providing a promising direction for improving system robustness. Our research underscores the value of careful model selection for real-world audio-based violence detection applications, recognizing the importance of an optimal trade-off between computational cost and performance, especially in resource-constrained scenarios. This research provides valuable insights for researchers and practitioners in developing more efficient, robust and accurate audio-based violence detection systems.

An Early Prediction Model for Estimating Bronchopulmonary Dysplasia in Preterm Infants

Introduction: Accurate assessment of the risk for bronchopulmonary dysplasia (BPD) is critical to determine the prognosis and identify infants who will benefit from preventive therapies. Clinical prediction models can support the identification of high-risk patients. In this study, we investigated the potential risk factors for BPD and compared machine learning models for predicting the outcome of BPD/death on days 1, 7, 14, and 28 in preterm infants. We also developed a local BPD estimator. Methods: This study involved 124 infants. We evaluated the composite outcome of BPD/death at a postmenstrual age of 36 weeks and identified risk factors that would improve BPD/death prediction. SPSS for Windows Version 11.5 and Weka 3.9 software were used for the data analysis. Results: To evaluate the combined effect of all variables, all risk factors were taken into consideration. Gestational age, birth weight, mode of respiratory support, intraventricular hemorrhage, necrotizing enterocolitis, surfactant requirement, and late-onset sepsis were risk factors on postnatal days 7, 14, and 28. In a comparison of four different time points (postnatal days 1, 7, 14, and 28), the day 7 model provided the best prediction. According to this model, when a patient was diagnosed with BPD/death, the accuracy rate was 89.5%. Conclusion: The postnatal day 7 model was the best predictor of BPD or death. Future validation studies will help identify infants who may benefit from preventive therapies and develop individualized care.

Infant death prediction using machine learning: A population-based retrospective study

BackgroundDespite declines in infant death rates in recent decades in the United States, the national goal of reducing infant death has not been reached. This study aims to predict infant death using machine-learning approaches. MethodsA population-based retrospective study of live births in the United States between 2016 and 2021 was conducted. Thirty-three factors related to birth facility, prenatal care and pregnancy history, labor and delivery, and newborn characteristics were used to predict infant death. ResultsXGBoost demonstrated superior performance compared to the other four compared machine learning models. The original imbalanced dataset yielded better results than the balanced datasets created through oversampling procedures. The cross-validation of the XGBoost-based model consistently achieved high performance during both the pre-pandemic (2016–2019) and pandemic (2020–2021) periods. Specifically, the XGBoost-based model performed exceptionally well in predicting neonatal death (AUC: 0.98). The key predictors of infant death were identified as gestational age, birth weight, 5-min APGAR score, and prenatal visits. A simplified model based on these four predictors resulted in slightly inferior yet comparable performance to the all-predictor model (AUC: 0.91 vs. 0.93). Furthermore, the four-factor risk classification system effectively identified infant deaths in 2020 and 2021 for high-risk (88.7%–89.0%), medium-risk (4.6%–5.4%), and low-risk groups (0.1), outperforming the risk screening tool based on accumulated risk factors. ConclusionsXGBoost-based models excel in predicting infant death, providing valuable prognostic information for perinatal care education and counselling. The simplified four-predictor classification system could serve as a practical alternative for infant death risk prediction.

Use of machine learning models for identification of predictors of survival and tumour recurrence in liver transplant recipients with hepatocellular carcinoma.

Hepatocellular carcinoma (HCC) is one of the leading indications for liver transplantation (LT) however, selection criteria remain controversial. We aimed to identify survival factors and predictors for tumour recurrence using machine learning (ML) methods. We also compared ML models to the Cox regression model. Thirty pretransplant donor and recipient general and tumour specific parameters were analysed from 170 patients who underwent orthotopic liver transplantation for HCC between March 2013 and December 2019 at the University Hospital Merkur, Zagreb. Survival rates were calculated using the Kaplan-Meier method and multivariate analysis was performed using the Cox proportional hazards regression model. Data was also processed through Coxnet (a regularized Cox regression model), Random Survival Forest (RSF), Survival Support Vector Machine (SVM) and Survival Gradient Boosting models, which included pre-processing, variable selection, imputation of missing data, training and cross-validation of the models. The cross-validated concordance index (CI) was used as an evaluation metric and to determine the best performing model. Kaplan-Meier curves for 5-year survival time showed survival probability of 80% for recipient survival and 82% for graft survival. The 5-year HCC recurrence was observed in 19% of patients. The best predictive accuracy was observed in the RSF model with CI of 0.72, followed by the Survival SVM model (CI 0.70). Overall ML models outperform the Cox regression model with respect to their limitations. Random Forest analysis provided several relevant outcome predictors: alpha fetoprotein (AFP), donor C-reactive protein (CRP), recipient age and neutrophil to lymphocyte ratio (NLR). Cox multivariate analysis showed similarities with RSF models in identifying detrimental variables. Some variables such as donor age and number of transarterial chemoembolization treatments (TACE) were pointed out, but these were not influential in our RSF model. Using ML methods in addition to classical statistical analysis, it is possible to develop sufficient prognostic models, which, compared to established risk scores, could help us quantify survival probability and make changes in organ utilization.

Comparing Machine Learning Models and Human Raters When Ranking Medical Student Performance Evaluations.

The Medical Student Performance Evaluation (MSPE), a narrative summary of each student's academic and professional performance in US medical school is long, making it challenging for residency programs evaluating large numbers of applicants. To create a rubric to assess MSPE narratives and to compare the ability of 3 commercially available machine learning models (MLMs) to rank MSPEs in order of positivity. Thirty out of a possible 120 MSPEs from the University of Central Florida class of 2020 were de-identified and subjected to manual scoring and ranking by a pair of faculty members using a new rubric based on the Accreditation Council for Graduate Medical Education competencies, and to global sentiment analysis by the MLMs. Correlation analysis was used to assess reliability and agreement between student rank orders produced by faculty and MLMs. The intraclass correlation coefficient used to assess faculty interrater reliability was 0.864 (P<.001; 95% CI 0.715-0.935) for total rubric scores and ranged from 0.402 to 0.768 for isolated subscales; faculty rank orders were also highly correlated (rs=0.758; P<.001; 95% CI 0.539-0.881). The authors report good feasibility as the rubric was easy to use and added minimal time to reading MSPEs. The MLMs correctly reported a positive sentiment for all 30 MSPE narratives, but their rank orders produced no significant correlations between different MLMs, or when compared with faculty rankings. The rubric for manual grading provided reliable overall scoring and ranking of MSPEs. The MLMs accurately detected positive sentiment in the MSPEs but were unable to provide reliable rank ordering.